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Abstract 

There has been increasing interest in modeling software product lines (SPLs) using architecture description languages (ADLs). 
However, sometimes it is required to reverse engineer an SPL architecture from a set of product architectures. This procedure 
needs to be performed manually as currently does not exist tool support to automate this task. In this case, verifying consistency 
between the product architectures and the reverse engineered SPL architecture is still a challenge; particularly, verifying 
component interconnection aspects of product architectures with respect to the commonality and variability of an SPL 
architecture represented in an ADL. Current approaches are unable to detect whether the component interconnections in a 
product architecture have inconsistencies with the component interconnections defined by the SPL architecture. To tackle 
these shortcomings, we developed the Ontology-based Product Architecture Verification (OntoPAV) framework. OntoPAV 
relies on the ontology formalism to capture the commonality and variability of SPLs architectures. Reasoning engines are 
employed to automatically identify component interconnection inconsistencies among SPL and product architectures. Our 
evaluation results show that our verifier has a high accuracy for detecting consistency errors and that it scales linearly for 
architectures from 1000 to 5000 architecture elements. 


Keywords Software product lines - Software architecture - SPL Verification - Architecture verification - Ontologies - 
Model-driven engineering 


1 Introduction 


Communicated by Ina Schaefer. 


Software product lines (SPLs) have proved to improve soft- 


EI Hector A. Duran-Limon ware quality and shorten costs and development time [1, 
hduran@cucea.udg.mx 2]. An SPL is a family of software systems that share a set 
Perla Velasco-Elizondo of common assets (commonly referred to as the SPL com- 
pvelasco@uaz.edu.mx monality) but also have some other characteristics that make 
Manuel Mora them different from each other (commonly named as SPL 
mmora@correo.uaa.mx variability). Feature modeling [3] is a technique that allows 
Maria E. Meda-Campana for specifying commonalities and variabilities in an SPL. A 
emeda@cucea.udg.mx feature represents an increment in product functionality [4]. 
Karina Aguilar Features are commonly represented in a tree structure called 
kaguilar@edu.uag.mx a feature diagram. A product configuration involves a selec- 
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tion of features from an SPL. Such a configuration represents 
a member of the product family. There are different types of 
relationships between a parent feature and a child feature [4]. 
A Mandatory relationship states that the child is included in 
all products belonging to the SPL to which the parent ele- 
ment belongs as well. An Optional relationship means that 
the child may or may not be part of a product. An Alternative 
relationship indicates that only one feature of a set of child 
features can be part of a product, whereas an OR relation- 
ship means that one or more features of a set of child features 
can be part of a product. Regarding cross-tree relationships, 
a requires relationship indicates that a feature requires the 
presence of another feature, whereas an excludes relation- 
ship excludes the presence of another feature. 

Product derivation concerns the process of building a 
product from an SPL at the implementation level, whereas 
product architecture derivation (also called architecture cus- 
tomization process) aims at producing a product-specific 
architecture from a generic product line architecture. The 
variability in the feature model is often expressed in terms 
of variation points (VPs), i.e., an area affected by variability, 
and systems options known as variants.! This variability can 
be also described in the software architecture and represented 
through different mechanisms. There has been increasing 
interest in modeling SPLs using Architecture description 
languages (ADLs), such as Koala [5] and PL-Xelha [6]. 
ADLs [7] represent formal notations for describing software 
architectures in terms of coarse-grained components and con- 
nectors. 

Importantly, consistency errors can take place when an 
SPL architecture is reverse engineered [8-10] from dif- 
ferent product architectures involving legacy applications. 
Although there have been some attempts to reverse engineer 
an SPL architecture [11-13], to the best of our knowledge, 
currently there are not approaches able to automatically 
extract ADL-based SPL architectures. Hence, the proce- 
dure to extract an SPL architecture from ADL-based product 
architectures needs to be carried out manually. Consistency 
errors (or inconsistencies) between an SPL architecture and 
a product architecture are those that make them incompati- 
ble so that the product architecture cannot be derived from 
the SPL architecture. For example, there is an inconsistency 
when a Mandatory connection defined in the SPL architec- 
ture is not present in a product architecture. Several efforts 
have focused on verifying consistency of feature models 
[14-18]. Other research has focused on verifying consis- 
tency of SPL architectures [19-24]. Only a few approaches 
have addressed the issue of checking consistency between 
a product architecture and an SPL architecture, but mainly 


' We distinguish the term variant from the term product variant. The 
former is an option of a variation point, whereas the latter is a specific 
derived product. 
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focusing on behavioral aspects [22, 23]. Hence, to the best 
of our knowledge there are not approaches able to verify the 
consistency of component interconnection aspects between 
a product architecture and the SPL architecture represented 
in an ADL. 

In this paper, we present the Ontology-based Product 
Architecture Verification (OntoPAV) framework, which ver- 
ifies consistency between the component interconnections 
of a product architecture and the component interconnec- 
tions defined in the SPL architecture. Our solution uses a 
language-independent approach to perform the verification 
process. We rely on the ontology formalism to capture the 
commonality and variability of an SPL architecture. In par- 
ticular, we employ Pellet [25], an ontology reasoner, to check 
consistency of product architectures. Although our approach 
involves the use of a formal method (i.e., ontology for- 
malism), the user does not require any knowledge about 
ontologies nor special training in any other technique from 
formal methods. We make use of model-driven engineering 
(MDE) techniques [26] to automate the generation process 
of the populated ontology. We have built a prototype to test 
our approach. Also, we developed an SPL of a web scholar 
system as a case example. 

We rely on the same meta-metamodel of the SPL Lan- 
guage defined in our previous work on product architecture 
derivation [6], which focused on automating the derivation of 
product architectures that are modeled with an ADL. Apart 
from the Reasoning Engine (a third party component), all 
the modules of the framework of the present work are differ- 
ent from those of the previous work.? In addition, this work 
is different from our previous work in two ways, mainly. 
First, in our previous work the generated populated ontol- 
ogy captures information about the SPL architecture and the 
feature tree, whereas in this work the produced populated 
ontologies include information about both the SPL architec- 
ture and a product architecture. Second, our previous work 
focused on generating a product architecture, whereas this 
work focuses on verifying the consistency of product archi- 
tectures of legacy systems with regard to a reverse engineered 
SPL architecture. 

The paper is structured as follows. Section 2 presents a 
motivating example. The OntoPAV framework is described 
in Sect. 3. Our ontology and the verification rules are defined 
in Sect. 4. Section 5 presents the Verification process to find 
the origin of errors. Evaluation results are shown in Sect. 6. 
Related work is included in Sect. 7. Finally, some concluding 
remarks are given in Sect. 8. 


? Even the Ontology Factory is different, as in the present work the 
populated ontology generated by this Factory states the valid Mandatory 
connections, OR connections, Alternative connections, and Optional 
connections as well as the requires/excludes restrictions. None of these 
restrictions are defined in the populated ontologies of our previous work. 
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2 Motivating example 


We use arunning example to illustrate the proposed OntoPAV 
framework throughout the paper. Our running example con- 
sists of a scholar system whose first products were not 
developed within an SPL. Although we do not have an SPL 
architecture of the system, there are a number of product 
architectures. Hence, the SPL architecture is reverse engi- 
neered. This is carried out manually given that currently there 
are not tool support for automating this task. We present 
two product architectures of the scholar system in Fig. 2. 
However, all the product architectures employed to reverse 
engineer the SPL architecture can be consulted in [27]. 

The scholar system allows for managing the information 
of students, teachers as well as the subjects and courses 
taught. The first configuration (i.e., product architecture 1), 
shown in Fig.2, can run a single term, whereas the second 
configuration (i.e., product architecture 2) can run multiple 
terms concurrently. The area of specialization is an Optional 
feature, given that it is present in the product architecture 1, 
but it is not present in the product architecture 2. Upon fin- 
ishing the courses included in a degree, the scholar system 
can offer different options for finishing such a degree, namely 
thesis, research article, and two or more years of professional 
practice. The product architecture | includes the former two, 
whereas the product architecture 2 includes the thesis and 
professional practice. 

A correctly derived SPL architecture of the scholar sys- 
tem is depicted in Fig.3 in PL-Xelha [6], which involves 14 
product architectures (and an SPL architecture with errors is 
shown in Fig. 4). Throughout the paper, we use PL-Xelha [6] 
only as a way to illustrate our approach since OntoPAV is 
language-independent, as mentioned earlier. Therefore, any 
modeling language that complies with the meta-metamodel 
defined in our previous work [6] can be used. Our meta- 
metamodel was defined in the metaobject facility (MOF).° 
Figure | depicts the meta-metamodel [6],* which defines 
the architectural commonality and variability. The former 
is defined in terms of Nodes and Connections. Nodes rep- 
resent fixed architectural elements, whereas a Connection 
is a binding between nodes. Also, Nodes may have Ports, 
which are interaction points. ProvidedPorts offer services, 
whereas RequiredPorts require them. Ports are inner ele- 
ments of both Nodes and variant elements. On the other 
side, Variability includes VariationPoints and Variants. The 
former permit the selection among different Variants. In addi- 
tion, VariantConnections are connections that bind Variants 
to VariationPoints. 

In PL-Xelha, there are two kinds of interfaces: provided 
and required. The former is represented by facets and offer 


3 http://www.omg.org/mof/. 
4 https://github.com/hduran-limon/copyRightMetaModel. 
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Fig. 1 Meta-metamodel 


services, whereas the latter is denoted by receptacles and 
requires services. The SPL architecture includes a num- 
ber of Mandatory components, such as c_student and 
c_teacher, which are represented with solid line boxes 
and are stereotyped as <<component>>, as shown in 
Fig.3. Variation points are denoted with dotted line boxes 
and are stereotyped as <<optionalComponent>>, 
which have one associated or more features. Variants are 
represented with solid line boxes and are stereotyped as 
<<variant>> and have defined the feature that repre- 
sents. A feature can be mapped to one or more compo- 
nents and a component can also define dependencies with 
other features; however, none of these characteristics are 
employed in order to simplify our example. vp_area rep- 
resents an Optional variation point, which is realized by 
the variant c_area when the feature Area is selected. 
An Optional variation point can have associated one or 
more Optional connections. An Optional connection is 
a connection (represented with a dotted line), which is 
optional. For instance, c_ui employs an Optional connec- 
tion to connect to vp_area, and the latter also employs an 
Optional connection to connect to c_course. In case the 
feature Area is selected, these two connections are real- 
ized in the corresponding product architecture; otherwise, 
both connections are removed. vp_term is an Alterna- 
tive variation point, which can be realized by the variant 
c_single_term in case the feature Singl]e is selected, 
whereas c_multiple_term instantiates this variation 
point when the feature Multiple is chosen. An OR varia- 
tion point is conformed by vp_professionalPractice, 
vp_article, and vp_thesis, and one or more of them 
can be instantiated by their associated variants. This depends 
on whether one or more of the following features are selected: 


> PL-Xelha is also able to denote connectors and Optional connectors, 
which in our running example are not used. 
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Fig. 2 Product architectures of the scholar system 
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Fig.3 SPL architecture of the scholar system without errors 
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Fig.5 Feature model tree of the scholar system 


ProfessionalPractice, Article, and Thesis. 
Lastly, the variant c_single_term requires c_area, 
whereas the variant c_multiple_term excludes it. The 
variability of the product family of the scholar system is 
shown in the feature tree of Fig. 5.° 

Now, consider the SPL architecture that is derived with 
errors (highlighted in red), as shown in Fig.4. Such errors 
may not be easy to identify without performing an auto- 
mated verification process. First of all, there are a number 
of Mandatory connections in the product architectures that 
do not appear in the SPL architecture, namely the con- 
nections between c_ui and c_course, and c_ui and 
c_teacher. Although in the latter case c_ui is con- 
nected to c_teacher, the facet interface name is wrong 
as it should be Iteacher instead of Isubject. Then, 
c_ui should be connected to vp_term, as shown in 
Fig. 3; however, this connection does not appear in Fig. 4. 
Also, c_degree should be connected to the following 
Alternative components: vp_professionalPractice, 
vp_article, and vp_thesis. Nevertheless, c_degree 
is not connected to any of them. On the other side, there 
are number of invalid connections in the SPL architec- 
ture, i.e., connections that are not defined in the cor- 
rectly derived SPL architecture, which involve the con- 
nections between c_degree and vp_term, c_ui and 
vp_article, c_ui and vp_thesis, and c_ui and 
vp_professionalPractice, as shown in Figs. 3 and 4. 
Furthermore, c_area should be required by c_single_ 
term; however, this component is required by c_multiple 
_term (see Figs. 3, 4). 

Consequently, we illustrate in the following sections, 
via our motivating example, how we can verify consis- 
tency between the component interconnections of a product 
architecture and the component interconnections of the SPL 
architecture. 


© Although the feature tree is not used by OntoPAV, we show it in order 
to improve the understandability of the motivating example. 


3 The OntoPAV framework 


The main elements of the OntoPAV framework, and their 
relationships, are presented in the collaboration diagram 
depicted in Fig.6. The ADL definitions of both the SPL 
architecture and the product architecture are the inputs of 
OntoPAV. These architectures can be defined in any ADL 
that complies with our meta-metamodel [6]. For instance, 
both the SPL architecture and the product architecture of the 
running example are defined in PL-Xelha, as shown in Figs. 3 
and 2,’ respectively. These ADL definitions are transformed 
to a populated ontology by the Ontology Factory. A popu- 
lated ontology includes instances of concepts, defined in the 
Ontology, such as component, connector, interface, feature, 
and variant. The populated ontology is used to verify consis- 
tency between the component interconnections of the product 
architecture and the component interconnections of the SPL 
architecture. In order to achieve language independence, a 
specific factory is generated for a specific SPL language. 

While the Ontology Factory is language-dependent (e.g., 
PL-Xelha requires a different implementation of this factory 
from that needed by Koala), the rest of the modules do not 
have any dependencies on the SPL language employed. 

The Ontology Factory asks the Verification Manager to 
verify the populated ontology (step 1). The Verification 
Manager is in charge of coordinating the checks performed 
and also delivers the verification result. For this purpose, 
the Verification Manager first makes a query to identify 
invalid connections (step 2). Second, the Verification Man- 
ager checks consistency of the populated ontology (this 
reflecting consistency of the product architecture, step 3); 
both cases with the aid of the Reasoning Engine. In case 
one or more inconsistencies are detected, the Verification 
Manager asks the Debugger to find the origin of such incon- 
sistencies (step 4). In order to find an inconsistency, the 
Debugger checks consistency on different subsets of the pop- 
ulated ontology to find the errors (step 5). The Debugger 
returns information with the origin of errors to the Verifica- 
tion Manager (step 6), which in turn informs the user about 
the verification result (step 7). 

Verifying component interconnections involves check- 
ing that in the product architecture Mandatory connections 
are present, Alternative connections have one and only one 
connection, OR connections have at least one connection, 
and Optional connections are present in case the associated 
Optional variation point is instantiated. Our framework also 
checks that invalid connections are not present in the product 
architecture. That is, OntoPAV checks that only component 
connections defined in the SPL architecture are present in 
the product architecture whereby components and connec- 


7 Only one architecture definition is taken as input of a product archi- 
tecture of OntoPAV. 


va Springer 


H. A. Duran-Limon et al. 


populate_ontology/( 
SPL_architecture, __-__-» 
product_architecture) 


‘Ontology 
Factory 


Software 
Architect 


| 


:Verification 
Manager 


2: query(q_invalid_conn, 
populated_ontology) | 


populated_ontology) 


1: verify(populated_ontology) 


<— 6: result(r) 


3: check_consistency( 


:Reasoning Engine :Debugger 


<¢ 
5: check_consistency(subset_populated_ontology) 


OntoPAV 


7: show_result(r) —--———+ > ‘ 
- wu ‘Result_ Viewer 


4: debug(populated_ontology) 


| 


Fig.6 Main elements of OntoPAV 


tors in the product architecture are interconnected in the right 
sequential order and with the right interface types. Lastly, 
our framework checks that requires/excludes relationships 
are met. 

OntoPAV can be used as follows. The software architect 
employs OntoPAV to check consistency between a reverse 
engineered SLP architecture and a product architecture. 
In case inconsistencies are detected, the software architect 
makes corrections to the SPL architecture and checks consis- 
tency again with the same product architecture. The architect 
makes corrections if inconsistencies are still detected, and so 
on. The same process is applied for each one of the product 
architectures from which the SPL architecture was reverse 
engineered. Future work considers verifying the SPL archi- 
tecture with the product architectures altogether in order to 
facilitate the process of making corrections to the SPL archi- 
tecture. 

In the following section, we present details of our ontology 
and the verification rules we employ. 


4 Verification rules and queries 


An ontology is a formal and explicit specification of a shared 
conceptualization of a domain [28, 29]. We make use of OWL 
[30] to represent ontologies. OWL is based on description 
logic (DL) [31], which is a family of logic-based knowledge 
representation formalisms. We use the Manchester OWL 
Syntax [30] to illustrate the OWL descriptions used for the 
verification. This syntax is easy to read and write and is 
mainly employed by many ontology editing tools such as 
Protégé [30]. 

An ontology employs individuals, classes, and object 
properties to describe the domain. Individuals, which are the 
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basic units in the domain, are instances of classes. Sets of 
individuals with similar characteristics conform to classes. 
Object properties represent binary relationships between 
individuals. Class constructors include the use of inter- 
section, union, and complement operations as well as the 
existential® and universal ° quantifiers, which in the Manch- 
ester OWL syntax are denoted by the keywords some and 
any, respectively. Class hierarchies can be defined by using 
the subclass property. A subclass is a subset of individuals 
of its parent class. A necessary condition!” is represented 
as an anonymous superclass, whereas a necessary and suffi- 
cient condition'' is represented as an equivalent class in the 
Manchester OWL Syntax format. Both conditions are also 
called restrictions. These and other features of OWL can 
be used to give a precise and unambiguous meaning to the 
descriptions of the domain. 

The elements of our ontology are represented in the SPL 
architecture metamodel depicted in Fig. 7. Hence, our ontol- 
ogy includes components, connectors, interfaces, and con- 
nections, which are used to represent product architectures. 
The ontology additionally employs Optional components, 
Optional connectors, variants, and features to represent SPL 
architectures. The former two are also called variation points. 
The population process of our ontology in OWL is based on 
the work of Wang et al. [32] where ontologies are used to 


8 This means that if a relation exists, at least one relation should exist 
with the specified class at the right side of the relation. 


» This means that ifa relation exists, none relation or at most one relation 
could exist with the specified class at the right side of the relation. 


10 Tf an individual is a member of this class, then it is necessary to 
fulfill this condition, but we cannot say that if an individual fulfills this 
condition, then it must be a member of this class. 


'l Tf an individual is a member of this class, then it is necessary to fulfill 
this condition, and if an individual fulfills this condition, then it must 
be a member of this class. 
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model feature relationships. We populate an ontology as fol- 
lows. 

First, we make use of disjoint OWL classes to represent 
components, connectors, Optional components, Optional 
connectors, interfaces, variants, and features. Each of these 
classes keep a subclass relationship with the OWL class 
Root. For instance, the component c_student is repre- 
sented as an OWL class named c_student. Variants keep 
a subclass relationship with their associated variation points 
(i.e., Optional components and Optional connectors). 

Second, each class defined previously!” has an equiv- 
alence statement, placed within its associated OWL class. 
Such an equivalence statement is a necessary and sufficient 
condition that binds the class to an existential restriction. For 
example, the existential restrictionhasC_student some 
c_student is defined with an equivalence statement in the 
OWL class c_student. 

Third, we represent component and connector connec- 
tions with existential statements as follows. Consider that 
the receptacle interface I; of element, is connected 
to the facet interface Ij; of elementg. Such a connec- 
tion is represented in the following form: element, — 
,isNextTo some elementg — I,. For example, the 
following statement specifies that the receptacle interface 
Istudent of the component c_ui is connected to the 
facet interface also named Istudent of the component 


c_student. 
(c_ui-IstudentIsNextTo some c_student 
- Istudent) 

Fourth, we included OWL restrictions in charge of associ- 
ating features with components and connectors. For instance, 
the OWL class named c_student has the existential state- 
ment hasStudent some Student. 

Fifth, we define a number of OWL restrictions for each 
class to specify Mandatory, OR, Alternative, Optional, and 
requires/excludes relationships (see below). In addition, we 
define a query mechanism, which is not part of a populated 
ontology, in charge of identifying invalid connections (see 
below). 


4.1 Verification rules 


The verification rules are defined in terms of OWL restric- 
tions and are used to detect inconsistencies between a product 
architecture and the SPL architecture. Next, we define one 
rule for product architectures, followed by six rules for the 
SPL architecture. 


Rule 1. Define restrictions for elements of the Product archi- 


tecture. An element can be a connection, a component, or a 


'2 This with the exception of the classes defined for features and inter- 
faces. 
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Fig.7 SPL architecture metamodel 


connector. We define an OWL restriction that specifies the 
elements that are present and those that are not present in the 
product architecture. 


Definition 1 Let elem;, elemg,... elem, be the elements 
that are present in the SPL architecture but are not present 
in the product architecture and elemp1, elempyg,... 
elemn+x be the elements that are present in both the SPL 
architecture and the product architecture. Such elements are 
specified with a necessary and sufficient condition in the 
OWL class named E_Product!? with the following form: 
not (elem,) andnot (elem,) andnot.... 
and not (elem,) and(elem,,,) and (elemm+2) 
and... and (elemy+x) 

In the case of our running example, below we specify an 
excerpt of the connections in the SPL architecture with errors 
(Fig. 4) that are not present in the product architecture | (see 
Fig. 2a), which represent the connections of c_degree with 
c_single_term and c_multiple_term, respectively. 
It is also specified that the component c_multiple_term 
is not present: 

not (c_degree-ItermIsNextTo some c 
_single_term-Iterm) 
not (c_degree-ItermIsNextTo some c_mu 
ltiple_term-Iterm) 
and not (hasC_multiple_term some c_mu 
ltiple_term) 


We present below an excerpt of the statement that specifies 
the connections that are present in the product architec- 
ture 1 (see Fig. 2a) involving the connections of c_ui with 
c_teacher and c_subject, respectively. It is also spec- 
ified that the component c_student is present: 


13 & Product is placed at the same level of Root in the populated 


ontology hierarchy. 
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(c_ui-IteacherIsNextTo some c_teacher 
-Iteacher) 
and (c_ui-IsubjectIsNextTo some c_sub 
ject-Isubject) 
and (hasC_student some c_student) 


Note that we have split the restriction statement into two 
parts for clarity only as both of them form a single restriction. 


Rule 2. Define restrictions for Mandatory connections of the 
SPL architecture. A Mandatory connection in an SPL archi- 
tecture takes place between two Mandatory elements and 
there is a one-to-one relationship. 


Theorem 2 Let P = {a},d2,...,dn} be the set of Mandatory 
connections defined in an SPL architecture and let Q = 
{b,,b9,...,Dm} be the set of Mandatory connections not defined 
in a product architecture, where Q C P. We associate the 
logical rule rulep with P and the logical rule ruleg with Q 
where: 

rulep > a, and a2 and... and ay and ruleg > 
not (bi) and not (bz) and... and not (b,) 
If there exists at least one connection a; € P and one con- 
nection b; € Q such that b; = a; and as a consequence 
Q # 9%, then a Mandatory condition is not met indicating 
that the product architecture is not consistent with the SPL 
architecture. In this case, rulep represents verification Rule 
2 and ruleg represents a simplification of verification Rule I 
involving only the set of Mandatory connections not defined 
in the product architecture and omitting any other type of 
connections. 


Proof If we assume that both rulep and ruleg are true and that 
there exists at least one connection a; € P and one connection 
bs € Osuch that b; = a; and as a consequence QO # 4, then 
we have that rulep = Q = G, which is a contradiction, and 
therefore, a Mandatory condition is not met. 


Definition 2 Let conn; be a Mandatory connection of the 
SPL architecture. There can be multiple Mandatory connec- 
tions, which we can represent as conn,, connz2, conns...., 
conny. Such connections are specified with a necessary con- 
dition in the OWL class Root with the following form: 

(conn,) and (conn,) and... and (conn,) 

Some of the Mandatory connections in our running 
example (see Fig.3) are the connections of c_ui with 
c_student and c_teacher, which are represented as 
follows: 


(c_ui-IstudentIsNextTo some c_student 
-Istudent) 
and (c_ui-IteacherIsNextTo some c_tea 
cher-Iteacher) 


Rule 3. Define restrictions for OR connections of the SPL 
architecture. OR connections involve variation points whose 
variant connections have an OR relationship. 
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Theorem 3 Let P = {aj,q9,...,dn} be the set of OR connec- 
tions defined in an SPL architecture and let Q = {b,,b3,...,bm} 
be the set of OR connections not defined in a product archi- 
tecture, where Q © P. We associate the logical rule rulep 
with P and the logical rule ruleg with Q where: 

rulep => a, OY a2OF.... OF an 

and ruleg > not (b,) and not (b2) and.... and 
not (b,) 
If there exists for each connection a; € P one connection 
b; € Q such that bj; = a; and as a consequence Q = P, 
then an OR condition is not met indicating that the product 
architecture is not consistent with the SPL architecture. Here, 
rulep represents verification Rule 3 and ruleg represents a 
simplification of verification Rule I involving only the set of 
OR connections not defined in the product architecture and 
omitting any other type of connections. 


Proof If we assume that both rulep and ruleg are true and 
that for each connection a; € P there exists one connection 
bs € Q such that b; = a; and as a consequence Q = P, then 
we have that rulep => Q 4 P, which is a contradiction, and 
therefore, an OR condition is not met. 


Definition 3 Let variantConn; be an OR connection 
of a variation point of the SPL architecture. There can 
be two or more OR connections associated with a vari- 
ation point, which we can represent as variantConn},, 
variantConn2, variantConn3,..., variantConny. 
Such OR connections are specified with a necessary con- 
dition in the OWL class Root with the following form: 

(variantConn;) or(variantConn,) or.... 
or (variantConn,) 

For example, the following statement specifies the OR 
relationships of the connections of c_degree with the 
instantiation of the variation points vp_professionalPr 


actice, vp_article, and vp_thesis (see Fig. 3). 
(c_degree-IdegreeOptionIsNextTo some 
c_professionalPractice-Iprofessional 
Practice) 
or (c_degree-IdegreeOptionIsNextTo some 
c_article-Iarticle)s 
or (c_degree-IdegreeOptionIsNextTo some 
c_thesis-Ithesis) 


Rule 4. Define restrictions for Alternative connections of the 
SPL architecture. Alternative connections involve variation 
points whose variant connections have an Alternative rela- 
tionship. 


Theorem 4 Let P = {a},d2,...,dn} be the set of Alternative 
connections defined in an SPL architecture and let Q; = 
{b,,b2,...,bm} be the set of Alternative connections not defined 
in a product architecture and Q> = {bm+1,bm+2,-..Dm+x } 
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be the set of Alternative connections defined in a product 

architecture, where P = Q; U Qo. We associate the logical 

rule rulep with Pandthelogicalruleruleg with Q where: 
rulep = (not (a,) and not (a2) and not 

(a3) and not... and not (ap-_j) and ay) 

or (not (a1) and not (az) and not (a3) and 


not... and (ap-1) and not ap) 
or (not (aj)and not (az)and not (a3)and 


not... and (ap-2)and not (ap-1)and not apy) 
(or (a,;) and not (az) and not (a3) and not 
..and not (ay_;) and not (an)) 

andruleg => not (bi) and not (b2) and not 
..andnot (b,) and (bm+1) and (bm+2) and... 
and (bm+x) 

In this case, rulep represents verification Rule 4 and ruleg 
represents a simplification of verification Rule 1 involving 
only the set of Alternative connections not defined and those 
that are defined in the product architecture and omitting any 
other type of connections. 


Theorem 4 a. /f we have Q) = 9 and that for each ai € P 
there is ab; € Q; such that b; = a; and as a consequence 
Q, =P, then we have that an Alternative condition is not met 
indicating that the product architecture is not consistent with 
the SPL architecture. 


Proof If we assume that both rulep and ruleg are true and 
Q> = and that for each a; € P there is ab; € Q; such that 
bj; = a; and as a consequence Q; = P, then we have that 
rulep => Q, 4 P, which is a contradiction, and therefore, an 
Alternative condition is not met. 


Theorem 4 b. /f Q; 4 P and there exists at least one pair of 
connections a; and a, € P and one pair of connections b; 

and bj € Q» such that b; =a; andb; =a, where i andk € 

{1,2,...,n},i<k,jand1 €{m+1,m+2,...,m+x} 

and 3 < 1 andas aconsequence | Q> | > 1, then we have that 
an Alternative condition is not met indicating that the product 
architecture is not consistent with the SPL architecture. 


Proof If we assume that both rulep and ruleg are true and Q; 
#% P and that there exists one pair of connections a; and a, 
€ P and one pair of connections b; and b; € Q> such that 
bj; =a; and bj = ax and as a consequence | Q> | > 1, then 
we have that rulep => | Q> | = 1, which is a contradiction, 
and therefore, an Alternative condition is not met. 


Definition 4 Let variantConn, bean Alternative connec- 
tion of a variation point of the SPL architecture. There can be 
two or more Alternative connections, which we can represent 


as variantConn;,variantConn2,variantConnsg...., 


variantConny. Such Alternative connections are spec- 
ified with a necessary condition in the OWL class of the 
variation point with the following form: 

(not (variantConn,;) and not (variant 
Conn) and not (variantConn3) and not... 
and not (variantConn,_,|) and variantConny) 
or (not (variantConn;,) and not (varia 
ntConn2) and not (variantConn3) and not 
. and (variant Conny,-1)and not variantConny) 
or (not(variantConn,)and not(variantConn2) 
and not (variantConn3) and not... 
and(variantConny-2)and not (variantConny-1) 


and not variantConn,) 

or ( (variantConn,) and not (variant 
Conn2) and not (variantConn3) and not... 
andnot (variantConn,.|) and not (variant 


Conny)) 

For example, the following statement specifies the Alter- 
native relationships of the connections of the component 
c_ui with the instantiation of the variants c_single_term 
and c_multiple_term (see Fig. 3). 

(not (c_ui-ItermIsNextTo some c_sing 
le_term-Iterm) 
and (c_ui-ItermIsNextTo some c_mult 
iple_term-Iterm)) 
or (not (c_ui-ItermIsNextTo some c_mult 
iple_term-Iterm) 
and (c_ui-ItermIsNextTo some c_single 
_term-Iterm)) 


Rule 5. Define restrictions for Optional connections associ- 
ated with an Optional variation point in the SPL architecture. 
This rule involves two parts. Part | regards a restriction indi- 
cating the Optional variation point is instantiated. Part 2 
defines the restrictions of the connections involved. There 
are four different cases for Part 2. This depending on 
whether the Optional connections are defined between (1) 
the Optional variation point and a Mandatory component, 
(2) two Optional variation points, (3) the Optional variation 
point and an OR variation point, and (4) the Optional varia- 
tion point and an Alternative variation point. Part 2 involves 
applying Rule 2 for cases one and two; and Rule 3 and Rule 
4 for case three and case four, respectively. Note that the 
clauses also include the case in which an Optional variation 
point is not instantiated prescribing the Optional connections 
must not be present. 


Definition 5.1 Let opConn,; be an Optional connection of 
the SPL architecture between an Optional variation point 
opVarPoint, and a Mandatory component. The instan- 
tiation of the Optional variation point instantiation; 
(Part 1) is specified together with such a connection (Part 2) 
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with a necessary condition in the OWL class Root with the 
following form: 

( (instantiation,;) and (opConn;) ) or 

( not (instantiation;)and not (opConn;)) 

There are two Optional connections associated with the 
Optional variation point vp_area that is instantiated with 
c_area in our running example (see Fig.3). These con- 
nections are between c_ui and vp_area and between 
vp_area and c_course. As an example, consider the con- 
straint for the latter: 

( (hasC_area some c_area)and 
(c_area-IcourseIsNextTo some c_course 
—Icourse) ) 
or 
( not (hasC_area some c_area)and not 
(c_area-IcourseIsNextTo some c_course 
—Icourse)) 


Definition 5.2 Let opConn, be an Optional connection of 
the SPL architecture between two Optional variation points, 
namely opVarPoint; and opVarPoint,, 1. The instan- 
tiation of the Optional variation points instantiation, 
instantiation,,, (Partl) are specified together with 
such a connection (Part 2) with a necessary condition in the 
OWL class Root with the following form: 

((instantiation;) and (instantiation;+1) 
and (opConn;)) or 


(not (instantiation;) or not (insta 
ntiation;+1) and 
(not (opConn;)) 


Definition 5.3 Let opConn,; be an Optional connection of 
the SPL architecture between an Optional variation point 
opVarPoint, and an OR variation point. There can be 
multiple Optional connections, which we can represent as 
opConn), opConn,, opConn3,..., opConn,. The instan- 
tiation of the Optional variation point instantiation; 
(Part 1) is specified together with such connections (Part2) 
with a necessary condition in the OWL class Root with the 
following form: 


( (instantiation; ) and ( not (not 
(opConn,) and not (opConn2) and not.... 
and not (opConn,,)))) 
or 
( not (instantiation;) and (not 
(opConn,) and not (opConn;) and not.... 


and not (opConn,))) 


Definition 5.4 Let opConn, be an Optional connection of 
the SPL architecture between an Optional variation point 
opVarPoint, and an Alternative variation point. There 
can be multiple Optional connections, which we can repre- 
sent as opConny,, opConn,, opConn3,..., opConny. The 


instantiation of the Optional variation point 
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instantiation, (Part 1) is specified together with such 
connections (Part2) with a necessary condition in the OWL 
class Root with the following form: 


( (instantiation; ) and ( (not 
(opConn,) and not (opConn 2) and not 
(opConn3) and not... and not (opConn,_;) 
and opConn,) 
or (not (opConn,) and not (opConn,) and 
not (opConn3) and not... and (opConn,-1) 


and not opConn,) 
or (not (opConn,)and not (opConn,z)and not 


(opConn3) and not...and(opConn,_,)and not 


(opConn,_1}) and not opConn,) 


or ( 


(opConn,;) and not (opConn,) and not 
(opConn3)and not... and not (opConn,-;) 
and not  (opConn,)))) 
or 
( not (instantiation;) and (not (opConn,) 
and not (opConn,) and not.... and not 


(opConn,)) ) 


Rule 6. Define requires restrictions in the SPL architecture. 
A requires restriction in an SPL architecture takes place 
between two or more elements whereby the former requires 
the presence of the latter elements. An element can be either 
a component or a connector. 


Theorem5 Let P = {aj,a2,...,dn} be the elements required 
by element e and let Q = {bj,b9,...,bm} be the set of elements 
not defined in a product architecture. We associate the logical 
rule rulep with Pandthelogicalruleruleg with Q where: 

rulep > a, andaz and... anday 

and ruleg > not (bi) and not (bz) and... and 
not (Dm) 
If there exists at least one element a; € P and one element 
b; € Osuch that bj = a; and as a consequence POF &, 
then a requires condition is not met indicating that the prod- 
uct architecture is not consistent with the SPL architecture. In 
this case, rulep represents verification Rule 6 and ruleg rep- 
resents a simplification of verification Rule I involving only 
the set of elements not defined in the product architecture and 
omitting any other type of elements. 


Proof If we assume that both rulep and ruleg are true and 
that there exists one element a; € P and one element bj € 
Q such that b; = a; and as a consequence PN O # Y, then 
we have that rulep > PN Q = G, which is a contradiction, 
and therefore, a requires condition is not met. 


Definition 5 Let element; be a required element of the 
element element, of the SPL architecture. There can 
be multiple required elements, which we can represent as 
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element), element2, element3,..., elementy,. Such 
required elements are specified with a necessary condition in 
the OWL class of the element element with the following 
form: 
(element;) and and 

(element,) 

For example, the following statement specifies the requires 
constraint of the component c_single_term (see Fig. 3) 
that requires the component c_area as follows: 


(hasC_area some c_area) 


(element,) and... 


Rule 7. Define excludes restrictions in the SPL architecture. 
An excludes restriction in an SPL architecture takes place 
between two or more elements whereby the former excludes 
the presence of the latter elements. An element can be either 
a component or a connector. 


Theorem 6 Let P = {a1,q9,...,an} be the elements excluded by 
an element e and let Q = {b,b2,...,bm} be the set of elements 
defined in a product architecture. We associate the logical 
rule rulep with P and the logical rule ruleg with Q where: 


rulep > not (a1) and not(az) and... and not 
(an) 
and ruleg => (b1) and(bz) and.... and (Bp) 


Tf there exists at least one element a; € P and one element b; 
€ QO such that bj = a; and as a consequence PO #Y, then 
an excludes condition is not met indicating that the product 
architecture is not consistent with the SPL architecture. In 
this case, rulep represents verification Rule 7 and ruleg rep- 
resents a simplification of verification Rule I involving only 
the set of elements that are defined in the product architec- 
ture. 


Proof If we assume that both rulep and ruleg are true and that 
there exists at least one element a; € P and one element b; 
€ Qsuch that b; = aj; and as aconsequence PN O # V, then 
we have that rulep > PM Q = Q, which is a contradiction, 
and therefore, an excludes condition is not met. 


Definition 6 Let element, be an excluded element of the 
element element, of the SPL architecture. There can 
be multiple excluded elements, which we can represent as 
element,, element2, element3,..., elementy. Such 
excluded elements are specified with a necessary condition in 
the OWL class of the element element, with the following 
form: 

not (element;)and not 
and not (element,) 


(element2)and... 


For example, the following statement specifies the excludes 
constraint of the component c_multiple_term (see 
Fig. 3) that excludes the component c_area as follows: 

not (hasC_area some c_area) 


4.2 Query invalid connections 


We employ a query that allows us to detect the presence of 
connections not defined in the SPL architecture (see step 2 
of Fig. 6). 

Query 1. Query the connections present in the Product 
architecture that are not part of the SPL architecture. Let 
spl_conn be the set of connections defined in the SPL 
architecture that represent the set of valid connections and 
p_conn be the set of the connections defined in the prod- 
uct architecture. This query retrieves the set of invalid 
connections {invalid_conn,, 
invalid_conng...., 


invalid_conng2, 
invalid_conn, } where 
invalid_conn; € p_conn and invalid_conn; ¢ 
spl_conn. 

An example of this query is presented in the next section. 


5 The verification process 


The verification process involves two phases. In the first 
phase, the Verification Manager employs Query | to iden- 
tify invalid connections. In the second phase, the Verification 
Manager checks whether the populated ontology is inconsis- 
tent in order to detect any errors related to Mandatory con- 
nections, OR connections, Alternative connections, Optional 
connections, and requires/excludes relationships. In case the 
populated ontology is inconsistent, the debugging process 
takes place. The Reasoning Engine is able to detect when a 
populated ontology is inconsistent; however, such an engine 
is unable to indicate what is causing the inconsistency. For 
instance, the Reasoning Engine is able to detect that there 
is an error when a Mandatory connection is missing in the 
product architecture. However, the Reasoning Engine does 
not provide any clue about the type of error nor the elements 
that may be involved in this error. The same happens with 
OR connections, Alternative connections, Optional connec- 
tions, and requires/excludes relationships. This issue makes 
it difficult to find the origin of errors in an inconsistent prod- 
uct architecture. In order to tackle this issue we make use 
of the Debugger, which, by employing multiple verification 
iterations of subsets of the populated ontology can find the 
origin of an error. Each verification phase is described further 
below. 

The verification information presents the connections in 
the SPL architecture that have inconsistencies. As men- 
tioned earlier, a connection is represented by element, — 
jisNextTo some elementg — Ij, where element, 
and elementg are components, and J; and Jj are their related 
interfaces. For example, the following verification result 
informs the users that there is an inconsistency with the 
connection between the receptacle interface Istudent of 
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the component c_ui and the facet interface also named 
Istudent of the component c_student. 

(c_ui-IstudentIsNextTo some c_student 
-Istudent) 


5.1 Identify invalid connections 


The Verification Manager employs Query | to identify the 
connections defined in the product architecture that are 
not specified in the SPL architecture. Regarding our run- 
ning example, the Verification Manager checks consistency 
between the product architecture 1 (Fig.2a) and the SPL 
architecture with errors (Fig.4). Hence, Query | returns the 
following invalid connections (i.e., connections in the prod- 
uct architecture | that are not present in the SPL architecture 
with errors): 
c_degree-IdegreeOptionIsNextTo some c 

_thesis-Ithesis 

c_degree-IdegreeOptionIsNextTo some c 
_article-Iarticle 
c_ui-ItermIsNextTo some c_single_term 
-Iterm 
c_ui-IcourseIsNextTo some c_course 


-Icourse 
c_ui-IteacherIsNextTo some c_teacher 
-Iteacher 

The former represents the connection between c_degree 
and c_thesis. In order to fix this error in the SPL 
architecture (Fig.4), c_degree needs to be connected 
to VP_thesis as c_thesis is a variant. The second 
case represents the connection between c_degree and 
c_article. Similarly, this error can be corrected by con- 
necting c_degree to VP_article. The third case is the 
connection between c_ui and c_single_term, which 
can be fixed by connecting c_ui to VP_term. The fourth 
case is the connection between c_ui and c_course, which 
can be easily repaired by connecting the former to the latter. 
The last case is a Mandatory connection whose facet interface 
is defined as Tsubject by the SPL architecture, whereas 
the product has Iteacher instead. This is simply rectified 
by renaming the facet interface of the SPL architecture to 
Iteacher. 


5.2 Identify errors with Mandatory, OR, Alternative, 
and Optional connections, and with 
requires/excludes relationships 


In case the populated ontology is inconsistent, the fol- 
lowing debugging process is carried out. First, the con- 
nection patterns of the SPL architecture are identified. In 
our running example, we have connection patterns such as 
IdegreeOptionIsNextTo, ItermIsNextTo, and 


TIcourselIsNextTo, among others. Such patterns involve 
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the receptacle interface that takes place in a connection. In 
this way the Debugger can group the set of connections errors 
that take place. !4 

For each connection pattern, i the Debugger constructs a 
node E_Debug_step_by_step;. Incase a node is incon- 
sistent, the Debugger raises an error showing the connections 
involved in such a node. In our running example (see Figs. 2a 
and 4), the Debugger identifies the following Mandatory con- 
nection error: 


Dee dninaecs c_ui-IteacherIsNextTo some c_te 
acher-Isubject 

The Debugger identifies that a Mandatory connection is 
missing from c_ui to c_teacher (line 1). Although in 
the product architecture, there is a connection between c_ui 
and c_teacher, the facet interface is named Iteacher 
in the product architecture 1 while the name defined in the 
SPL architecture with errors is Tsubj ect. This error is also 
detected by Query 1!°. This error was already fixed above. 

The Debugger identifies the following OR connection 
error: 


Doane guiets c_ui-IdegreeOptionIsNextTo some 
c_article-Tarticle, c_ui-IdegreeOption 
IsNextTo some c_professionalPractice 
-IprofessionalPractice, c_ui-Idegree 
OptionIsNextTo some c_thesis-Ithesis 

There is an OR connection error between c_ui and 
the following three variation points (line 2): vp_profe 
ssionalPractice, vp_article, and vp_thesis. 
In the product architecture 1, c_ui is not connected to 
any of the three variants related to these variation points, 
namely c_professionalPractice, c_article, or 
c_thesis (see Fig.2a), while at least one connection 
should be present according to the SPL architecture with 
errors (see Fig.4). This error can be simply repaired by 
removing the OR connection from c_ui in the SPL archi- 
tecture (Fig. 4). 

In addition, the Debugger identifies the following Alter- 
native connection error: 

Bhscase aes c_degree-ItermIsNextTo some 
c_multiple_term-Iterm, 
c_degree-ItermIsNextTo some 
c_single_term-Iterm 

There is an Alternative connection error between the 
component c_degree and the variation point vp_term 
(line 3). The reason of this error is c_degree is not 


14 We assume that each connection between two elements has a dif- 
ferent connection pattern. That is, a receptacle interface name must be 
a singleton in the SPL architecture, i.e., there cannot be two or more 
receptacle interfaces with the same name unless they are connecting 
other instances of the same two elements and in the same order. 


'5 Query 1 only detects invalid connections, but in this case it happened 
that this invalid connection (one interface name is not consistent) is also 
a Mandatory connection between the two components. 
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connected to any variant of this variation point, namely 
c_single_termand c_multiple_term (as shown in 
Fig. 2a) while the SPL architecture with errors indicates that 
c_degree should be connected to one of them (as shown in 
Fig. 4). This error can be easily fixed by removing the Alter- 
native connection from c_degree in the SPL architecture 
(Fig. 4). 

The Debugger also identifies the following Optional con- 
nection error: 


A ene c_area-IsubjectIsNextTo some 
c_subject-Isubject 

There is an Optional connection error between the 
Optional variation point vp_area and the component 
c_subject (line 4). Given that the Optional feature Area 
was selected, the Optional variation point should be instan- 
tiated with the component c_area as well as a connec- 
tion between the components c_area and c_subject, 
c_area and c_course, and c_ui and c_area but the 
connection between c_area and c_subject is not present 
(as shown in Fig.2a); however, the latter connection is 
included in the SPL architecture with errors (as shown in 
Fig. 4). This error can be simply corrected by removing the 
Optional connection between vp_area and c_subject 
in the SPL architecture (Fig. 4). 

Lastly, the Debugger identifies the following requires con- 
straint is not met: 


Diy aes hasC_area some c_area 

According to the SPL architecture with errors, when the 
variant c_Single_termis selected, it excludes the variant 
c_area to be selected (as shown in Fig. 4). However, in the 
product architecture 1 c_single_term is present as well 
as C_area (as shown in Fig. 2a). This error can be fixed by 
first removing the requires constraint in the SPL architec- 
ture (Fig.4). Then, we need to reverse engineer the correct 
requires constraint, if any, from the product architectures and 
place it in the SPL architecture. 


6 Evaluation 


In this section, we describe the evaluation of both the accu- 
racy and scalability of our approach. We also present a 
discussion of the results and discuss the limitations and 
threats to validity of our work. 


6.1 Prototype implementation 


We used a tool [6] to visually edit both the SPL architec- 
ture and the product architecture in PL-Xelha. Regarding the 
implementation of the Ontology Factory, we used Acceleo 
(33] to transform the PL-Xelha models of the SPL archi- 
tecture and product architecture to text in Turtle [34]. The 
Turtle syntax is supported by most ontology-based rea- 


soning engines and its syntax is less verbose and more 
user-friendly than other formats. As mentioned earlier, we 
used Pellet [25] as the Reasoning Engine. The Verification 
Manager and the Debugger were implemented in Java. Lastly, 
Query | was implemented in Java as part of the Verifica- 
tion Manager. As mentioned earlier, the Ontology Factory 
is language-dependent (e.g., PL-Xelha requires a different 
implementation of this factory from that needed by Koala). 
However, the rest of the modules do not have any dependen- 
cies on the SPL language employed. 


6.2 Accuracy evaluation 


We used mutation testing [35] to evaluate the accuracy of our 
verification approach. Mutation testing has been employed 
for evaluating the adequacy of test suites [36-38] and also 
for guiding the generation of test cases [38-40]. Mutation 
testing has also been used to test feature models in a soft- 
ware product line [41, 42] as well as a means to generate 
test configurations for an SPL [43] and assess the ability of 
test suites to detect errors in feature models [44]. Mutants 
are software artifacts that have artificial errors injected. Such 
artificial errors are called mutations. Mutation operators are 
mutation rules used to inject mutations. In case the test suit is 
able to detect a mutation (i.e., a test case fails), it is said that 
a mutant is killed. Equivalent mutants are those whose func- 
tionality is equivalent to the original artifacts. The ratio of 
mutants detected by the test cases is called the mutation score. 
In this paper, the mutation score represents the accuracy of 
our verifier to detect errors. We calculated the mutation score 
as follows: 

m_score = mutants_killed / (mutants 

- e mutants) * 100 

where m_score is the mutation score, mutants_killed 
are the number of mutants killed, mutants are the num- 
ber of selected mutants, and e_mutants are the number of 
equivalent mutants. Equivalent mutants are subtracted from 
the number of mutants since equivalent mutants are not able 
to produce errors. 

We employed mutation testing to systematically derive 
test cases for our running example.!© Based on the test 
cases generated, we evaluated the ability of our verifica- 
tion approach to detect inconsistencies between an SPL 
architecture and a product architecture. We achieved this by 
conducting the following steps: 


'6 The number of product architectures of the SPL architecture 
employed by the evaluation is 15. This given that our running example 
was extended with an Optional connection between the Optional com- 
ponent vp_termand the Optional component vp_area (see mutation 
operators 24 and 28). 
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1. Define mutation operators. In order to define mutation 
operators, we identified what can be changed in an SPL 
architecture. These operators were defined based on the 
operations add, remove, and change, which are the basic 
operations needed to introduce changes. Then we applied 
these operations to the different elements of an SPL 
architecture such as Mandatory component, Optional 
component, OR group, and Alternative group. A partial 
list of the mutation operators we applied to the SPL archi- 
tecture of our running example (see Fig. 3) is presented 
in Table |. The full list of the mutation operators is given 
in “Appendix A” in Table 7. 

2. Generate the mutants. We manually applied the mutation 
operators to the SPL architecture of our running example 
to obtain the mutants. The generated mutants are SPL 
architectures with changes introduced. 

3. Generate test cases. We generated a test case for each 
mutant. Thus, a test case is a product architecture that 
is consistent with a mutant (1.e., a modified SPL archi- 
tecture). There may be multiple test cases associated 
with a single mutant. Some of these test cases could be 
mutant killers and some others could be not. We randomly 
selected only one mutant killer. 

4. Evaluate the verifier with the test cases. The steps to 
evaluate the verifier are as follows. First, we verify the 
test cases against the mutants where no verification errors 
should be raised. Contrarily, the test case is erroneous 
and must be corrected until no errors are detected by the 
verifier. Second, the test cases are verified with respect 
to the original SPL architecture of the running example. 
In this case, it is expected that the verifier detects one or 
more errors related to the changes made by the associated 
mutation operator. 


The results of the mutation testing are shown in Table 2. 
The first column shows the total number of mutants that 
can be produced by applying each mutation operator. The 
second column presents the number of mutants that were 
actually generated. For example, in the case of the opera- 
tor “12. Change the instantiation relationship of an Optional 
component,” there are 30 possible mutants. However, we only 
generated 20% of them (i.e., 6) to reduce the effort of manu- 
ally generating them. We also selected 20% or higher of the 
mutants in the case of the operators 17, 18, 19, 20, 31, and 
33. It has been shown elsewhere [45] that randomly select- 
ing 20% of the mutants results in a fault loss of only about 
16%. The percentage of selected mutants is shown in the 
third column. Then, the fourth column shows the cases when 
a mutant is killed. This happens when a test case fails, as 
earlier mentioned. Lastly, the fifth column shows the equiv- 
alent mutants, which are those mutants that are equivalent 
to the original SPL. All the mutants generated by the oper- 
ators 6, 7, 10, and 11 are equivalent mutants. In the case 
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of operator 8, only one mutant out of five is mutant equiv- 
alent. In the case of operator 12, there are two mutants out 
of six that are mutant equivalent. We obtained 116 mutants 
killed, which is exactly the same number that results from 
subtracting the number of equivalent mutants (i.e., 14) to 
the number of selected mutants (i.e., 130). As a result, our 
verifier obtained a mutation score of 100%. 


6.3 Scalability evaluation 


We carried out a number of experiments to test the scalability 
of our approach. Each experiment was repeated 30 times and 
the average execution time of these runs was considered. 

We carried out our experiments on a MacBook Pro Retina 
Intel Core 17 at 2.5 GHz with 6 MB of L3 cache memory 
and 16 GB of RAM running Mac OS X, version 10.13.2. 
The scalability of the prototype was evaluated for different 
sizes of SPL architectures. In our experiment, we tested the 
scalability of the Ontology Factory, Verification Manager, 
and Debugger. The Reasoning Engine was indirectly tested 
by the latter two as these modules perform queries over the 
ontologies. 

In order to test the scalability of our framework, we imple- 
mented a Java program in charge of generating random SPL 
architectures. We chose using random architectures as no 
industrial SPL architectures are freely available and with ran- 
dom architectures we have more flexibility to decide the size 
of the SPL architecture to test. This program also gener- 
ates a set of product architectures for each SPL architecture 
that corresponds to a number of random selections from the 
set of valid configurations. Errors were randomly introduced 
in the generated product architectures, which consisted of 
removing an arbitrary Mandatory connection and an arbi- 
trary Alternative connection from the product architectures. 
In this way, the Debugger was forced to check the validity of 
Mandatory, OR, and Alternative connections. 

In addition, the architecture generator program can pro- 
duce SPL architectures of different sizes that follow a 
proportion similar to our running example and involves the 
same number of Alternative variation points as the number 
of Mandatory elements and requires/excludes relationships 
altogether. Such relationships are approximately the fifth part 
of Mandatory elements. There are twice as many OR vari- 
ation points as there are Alternative variation points and 
there are two to seven variants associated with each vari- 
ation point (the specific number of variants was randomly 
selected). Alternative variation points were connected con- 
secutively in line (one to another) in order to test the worst 
case scenario (i.e., acase with higher computing complexity). 
This is because Rule 3 statements grow exponentially as the 
number of variants increases; hence, scalability is affected 
accordingly (see below). 
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Table 1 Partial list of mutation operators 


Operator 


1. Remove a Mandatory component 

2. Change a Mandatory component 
to an Optional component 

3. Change the name of a 
Mandatory component 

4. Add an Optional component to 
an OR group 

5. Add an Optional component to 
an Alternative group 

6. Remove an Optional component 
from an OR group 

7. Remove an Optional component 
from an Alternative group 

8. Remove an Optional component 
and its associated variants 

9. Change the name of a variant 

10. Change an Optional component 
to a Mandatory component 

11. Change the name of an 
Optional component 

12. Change the instantiation 
relationship of an Optional comp 

13. Remove an OR group 


relationship 


14. Change OR group relationship 
to Alternative group relationship 

15. Remove an Alternative group 

relationship 

16. Change Alternative group 
relationship to OR group relation 

17. Add a connection between two 
Mandatory components 

18. Add a connection between a 
Mandatory component and an 

Optional component 

19. Add a connection between an 
Optional component and a 
Mandatory component 

20. Add a connection between two 
Optional components 

21. Remove a connection between 
two Mandatory components 

22. Remove a connection between 


a Mandatory comp and Op comp 


Example 


Remove the Mandatory component c_student from the SPL architecture 
Convert the Mandatory component c_student to an Optional 

component in the SPL architecture 
The name of the Mandatory component 

c_student is changed to c_studentV2 


The Optional component vp_area is disconnected from c_ui and connected 


to the receptacle TdegreeOption of the component c_degree as another degree option 


The Optional component vp_area is removed and its associated variant c_area is 
added as an alternative of the Optional component vp_term 

Remove both the Optional component vp_thesis and its associated variant 
c_thesis from the OR group in the SPL architecture 

The variant c_single_term is removed from the Alternative group in the SPL 
architecture 

Remove both the variation point vp_area and its associated variant c_area 
from the SPL architecture 

Change the name of the variant c_thesis by c_thesisv2 

The Optional component vp_area and its associated variant c_area are transformed to 
a Mandatory component and the requires/excludes relationships are eliminated 

The name of the variant c_area is changed to c_areaV2. Another 
example is changing the variant c_thesis to c_thesisV2 

The variants c_thesis and c_article are interchanged so that now they instantiate 
the Optional components vp_article and vp_thesis, respectively 


The Optional components vp_professionalPractice, vp_article 


and vp_thesis are removed along with their associated variants c_professionalPractice 


c_article, and c_thesis 

The OR group relationship of our running example is transformed to an Alternative 
group relationship 

The Optional component vp_term is removed along with its variants 

c_single_termand c_multiple_term 

The Alternative group relationship of our running example is transformed to an OR 
group relationship 

A receptacle of the component c_student is connected to a facet of the component 
c_degree 

A receptacle of the component c_course is connected to a facet of the Optional 


component vp_area 


Connect a receptacle of the Optional component 
vp_area to the facet of the component 
c_course 

Connect a receptacle of the Optional component vp_area to a facet of the Optional 
component vp_term 


The connection between the components c_ui and c_course is removed 


The connection between the component c_ui and the Optional component vp_area 


is removed 
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Table 1 continued 


Operator Example 


23. Remove connection between Op 
comp and a Mandatory comp 
24. Remove a connection between 


two Optional components 
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The Optional connection between the Optional component vp_area and the 
component c_course is removed 
Given that our running example does not count with such a type of connection 


we added an Optional connection between the Optional component vp_term 


and the Optional component vp_area, and we considered this as the original SPL 


architecture for testing the operator. Then, after applying this operator, this 


connection is removed to obtain the mutant 


25. Change the connection of a 
Mandatory receptacle with another 


Mandatory receptacle 


The receptacles Isubject and Icourse of the component c_ui 


are connected to the facets Icourse and Isubject, respectively 


The experiments we carried out involved different sce- 
narios, whose details are presented below. In scenario 1, we 
tested the scalability of our approach. In Table 3, we show the 
number of architecture elements we used. We can see that the 
experiments in this scenario involve SPL architectures start- 
ing with 57 architecture elements up to 285 elements. The 
scalability behavior of the Ontology Factory is almost lin- 
ear as shown in Fig. 8. Figure9 shows that the performance 
degradation of the Verification Manager increases in a linear 
way demanding up to 10 s for 285 elements. However, the 
performance of the Debugger degrades exponentially as more 
elements are included in the SPL architecture demanding up 
to 463 s for 285 elements, as shown in Fig. 10. 

Given that we found that the Debugger degrades rapidly 
we adopted an strategy that allowed our approach to handle 
larger architectures. This strategy involves making sub- 
partitions of the SPL architectures together with their asso- 
ciated product architectures into smaller sub-architectures. 
We generated such sub-architectures manually. We con- 
sidered two constraints for maintaining the verification of 
partitioned architectures consistent. First, it was considered 
that two adjacent sub-architectures cross-cut each other to 
avoid loosing information, where the second sub-architecture 
includes as its first element the last element of the first sub- 
architecture. Such an element can be a variation point, a 
component, or a connector, in the case of an SPL architec- 
ture. In the case of the product architecture, this element can 
be a component or a connector. Second, component and con- 
nector connections cannot cross-cut two sub-architectures. 
In addition, the maximum size of the sub-architectures was 
set by determining the average rate of change of the perfor- 
mance degradation. We calculate the average rate of change 
by dividing the change in the execution time by the change 
in the number of architecture elements. We defined a maxi- 
mum rate of change of around one in order to keep a linear 
performance degradation when the size of the SPL archi- 
tecture scales up. A rate of change greater than one means 
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that the execution time grows more rapidly than the num- 
ber of architecture elements. We found, in the case of our 
testbed platform, that SPL sub-architectures of up to 216 
were suitable. This because as a base line we had 62 ele- 
ments which took 13.70 s. Then we got the average rate of 
change of the base line with respect to different amounts of 
elements. We obtained an average rate of change of 0.697 
for 216 elements. The rate of change was larger than one for 
larger sub-architectures. 

Therefore, in scenario 2, we employed the partitioning 
strategy to test the scalability of our approach with larger 
architectures. In Table 4 we show the number of architec- 
ture elements we used for this scenario, which ranges from 
1026 elements to 5016 elements; and we considered a vary- 
ing number of requires/excludes relationships, Mandatory 
elements, variation points, and variants ranging from 9 to 44, 
45 to 220, 162 to 792, and 810 to 3960, respectively. We 
can see that the Ontology Factory, the Verification Manager, 
and even the Debugger exhibit a linear scalability behavior in 
scenario 2, as depicted in Figs. 11, 12, and 13, respectively. In 
the case of the Debugger experiments (Fig. 13), we reduced 
the number of repetitions to five given that in some cases 
each execution took around one hour. Based on the central 
limit theorem and given that in this experiment scenario the 
sampling distribution was nearly normal, we considered the 
sample size (i.e., five runs) to be large enough. 

In scenario 3, we tested how the performance of the 
Debugger degraded for two Alternative variation points 
(VPs) interconnected and a Mandatory component con- 
nected to one of the VPs when the amount of variants 
increases. We performed this test given that the computa- 
tional complexity of Rule 3 statements, which define the 
restrictions for Alternative connections, grows exponentially 
as the number of variants increases, specially when two 
Alternative VPs are interconnected. We focused only on the 
degradation of the Debugger since it suffers a much higher 
degradation than both the Ontology Factory and the Verifi- 
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Table 2 Results of mutation testing 


Operator Mutants Selected % Selected Mutants Equivalent 
Mutants Mutants Killed Mutants 

1. Remove a Mandatory component 6 6 100 6 0 
2. Change a Mandatory comp to an Optional comp 6 6 100 6 0 
3. Change the name of a Mandatory component 6 6 100 6 0 
4. Add an Optional component to an OR group =) 3 100 3 0 
5. Add an Optional component to an Alternative group 1 1 100 1 0 
6. Remove an Optional component from an OR group 3 3 100 0 3 
7. Remove an Optional comp from an Altern group 2 2 100 0 2 
8. Remove an Optional comp and its assoc variants 5 5 100 4 1 
9. Change an Optional comp to a Mandatory comp 6 6 100 6 0 
10. Change an Optional comp to a Mandatory comp 1 1 100 0 1 
11. Change the name of an Optional component 5 5 100 0 5 
12. Change the instantiation relationship of an 

Optional component 30 6 20 4 2 
13. Remove an OR group relationship 1 1 100 1 
14. Change an OR group relationship to an 

Alternative group relationship 1 1 100 1 0 
15. Remove an Alternative group relationship 1 1 100 1 
16. Change an Alternative group relationship 

to an OR group relationship 1 1 100 1 0 
17. Add a connection between two Mandatory comp 24 5 20.83 5 
18. Add a connection between a Mandatory comp 

and an Optional component 27 6 22.22 6 0 
19. Add a connection between an Optional comp 

and a Mandatory component 25 P 20 5 0 
20. Add a connection between two Optional comp 20 4 20 
21. Remove a connection between two Mandatory comp 6 6 100 6 
22. Remove a connection between a Mandatory comp 

and an Optional component 1 1 100 1 0 
23. Remove a connection between an Optional comp 

and a Mandatory component 1 1 100 1 
24. Remove a connection between two Optional comp 1 1 100 1 0 
25. Change the connection of a Mandatory receptacle 

with another Mandatory receptacle 5 > 100 5 0 
26. Change the connection direction between two 

Mandatory components 6 6 100 6 0 
27. Change the connection direction between a 

Mandatory component and an Optional component 5 5 100 5 0 
28. Change the connection direction between two 

Optional components 1 1 100 1 0 
29. Change the name of a receptacle interface 8 8 100 8 0 
30. Change the name of a facet interface 8 8 100 8 0 
31. Add a requires relationship 27 6 22.22 6 0 
32. Remove a requires relationship 1 1 100 1 0 
33. Add an excludes relationship 27 6 22.22 6 0 
34. Remove an excludes relationship 1 if 100 1 0 
Total 272 130 - 116 14 
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: : Requires/excludes relationships Mandatory OR VPs Alternat. VPs Variants Total 
elements in Scenario | 
0 3 6 3 45 57 
1 > 12 6 90 114 
1 8 18 9 135 171 
2 10 24 12 180 228 
2 13 30 15 225 285 
fable + Sie at baie : Requires/excludes relationships Mandatory OR VPs Alternat. VPs Variants Total 
architectures in Scenario 2 
9 45 108 54 810 1026 
17 88 210 105 1575 1995 
26 133 318 159 2385 3021 
35 178 426 213 3195 4047 
44 220 528 264 3960 5016 
50 
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Fig.8 Scenario 1, scalability of the Ontology Factory 


cation Manager, as shown in scenario 1. Table 5 depicts the 
performance of the Debugger degrades rapidly as the number 
of variants increases. In this case, we also reduced the num- 
ber of repetitions to five given that in the case of 30 variants 
each execution took more than one hour. We obtained the 
average rate of change of the base line with respect to differ- 
ent amounts of variants. As a base line, we took the first row 
of the table. In addition, we tested the performance degrada- 
tion of the Debugger for a Mandatory component connected 
to an OR variation point. Table 6 shows that in this case, a 
higher number of variants can be handled by the Debugger. 


6.4 Discussion 


The accuracy evaluation results have shown that our verifier 
has a high accuracy for detecting errors in product archi- 
tectures (i.e., elements in a product architecture that are not 
consistent with an SPL architecture). We defined 34 mutant 
operators and evaluated 130 mutants (i.e., 130 modified 
SPLs) where 116 mutants were killed and 14 mutants were 
mutant equivalent. Our accuracy evaluation obtained a muta- 
tion score of 100%, which indicates that all errors in the test 
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Fig.10 Scenario 1, scalability of the Debugger 


cases were correctly detected and the sources of errors were 
correctly identified by the Debugger. The verifier was unable 
to detect errors only in the cases the mutants were mutant 
equivalent. This is the expected behavior of the verifier since 
the products of a mutant equivalent are consistent with the 
original SPL. This was the case of removing a component 
from an OR group (operator 6), removing a component from 
an Alternative group (operator 7), or changing the name of 
an Optional component (operator 11), which only shrink the 
configuration space but do not generate test cases with errors. 
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Fig.11 Scenario 2, scalability of the Ontology Factory 
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Fig.12 Scenario 2, scalability of the Verification Manager 
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Fig. 13 Scenario 2, scalability of the Debugger 


As an example, consider the mutant operator “7. Remove an 
Optional component from an Alternative group” that gener- 
ates mutants that are mutant equivalent. In case we apply this 
operator and remove the variant c_single_term from the 
SPL, all products of the mutant will still be consistent with 
the original SPL. On the other hand, the selected variants 
of a product configuration that are not directly related to 
the change of a mutant do not affect the verification result. 
Therefore, we did not create all possible test cases, rather, 
we randomly selected a test case that is directly related to the 
change made to the mutant; this to make sure that this test 
case is a mutant killer. For instance, operator 2 regards chang- 
ing a Mandatory component to an optional component such 


Table 5 Scenario 3a. Scalability of the Debugger for two Alternative 
VPs interconnected 


Variants per Alternative VPExecution Time (s)Average Rate of Change 


10 3.12 - 

12 5.28 1.07 
15 21.51 3.67 
20 86.36 8.32 
25 286.08 18.86 
30 792.46 39.46 


Table 6 Scenario 3b. Scalability of the Debugger for an OR VP con- 
nected to a Mandatory Component 


Variants perOR VP _ Execution time (s) Average rate of change 


100 1.92 - 

200 5.51 0.03 
300 10.42 0.04 
400 16.78 0.04 
500 26.32 0.06 


as converting the Mandatory component c_student to an 
optional component. In this case, we select a product that 
does not include c_student for it to be a mutant killer. In 
other cases, all products are mutant killers. For example oper- 
ator 15 involves removing an Alternative group relationship. 
In this case, all products of the mutant are mutant killers. 
This is because in all products both connections c_ui to 
c_single_term and c_ui to c_multiple_term are 
missing. Nevertheless, the important point is finding a mutant 
killer and not every possible mutant killer. This is because 
one mutant killer is enough to test the same type of point of 
failure. 

Regarding the scalability evaluation, the first scenario is 
useful to find out the maximum number of architectural 
elements our approach is able to handle on a particular 
hardware platform without degrading by partitioning it into 
sub-architectures, where each sub-architecture does not sur- 
pass such a number. The second scenario shows that the 
partition approach is useful to keep the linear performance 
behavior. The third scenario helps us to find out the max- 
imum number of variants an Alternative variation point is 
able to handle on a particular hardware platform. We can 
observe that in both scenario | and scenario 2 the Debugger 
takes much more time to execute than both the Ontology Fac- 
tory and the Verification Manager. The experimental results 
in scenario | show that the Debugger does not scale well 
for SPL architectures that include more than 225 elements. 
The expressiveness obtained by description logics is achieved 
at the expense of higher computational complexity that is 
present in current ontology reasoning engines [46]. This issue 
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is exacerbated by the fact that the Debugger employs multiple 
queries and consistency checks in the debugging process. The 
Debugger employs a consistency check for each component 
connection pattern in the SPL architecture. We addressed this 
scalability issue by using a partitioning strategy, whereby we 
partition large SPL architectures and product architectures 
into small sub-architectures. We have shown that by using 
this partitioning strategy, the scalability of our framework 
is linear. For instance, in the case of an SPL architecture 
of 1026 elements, the Debugger takes 603 s, whereas in the 
larger case with an SPL of 5016 elements the Debugger takes 
3525 s. 

The third scenario shows that Alternative variation points 
face scalability problems. The restrictions of Alternative 
elements are defined according to Rule 3. The scalability 
problems are due to the fact that restrictions of n Alterna- 
tive connections require n statements each one involving 
n terms to define an Alternative variation point restriction; 
hence, the number of terms needed is n”. In the case of inter- 
connecting an Alternativevariationpointvp, to a variation 
point vp», where vp, is either another Alternative variation 
point or an OR variation point whereby vp, and vp, have 
n and m variants, respectively, we have n*m possible con- 
nections. Therefore, restrictions of Alternative connections 
require n*m statements, where each statement involves m 
terms; hence, the total number of terms needed is n*m2. In 
order to avoid scalability problems we suggest that those 
Alternative variation points that are connected to another 
variation point be partitioned into a sub-architecture when 
the number of variants of the Alternative variation point has 
an average rate of change considerably higher than 1, which 
in our platform happens from 15 variants (see Table 5). Then, 
we suggest limiting the maximum number of variants asso- 
ciated with an Alternative variation to the point where the 
Debugger exhibits a rate that is close to one (which in our plat- 
form happens to be around 12 variants as shown in Table 5). 
Although our approach is able to handle only a short number 
of variants of Alternative variation points, OntoPAV is able to 
mange a larger number of variants in the case of a Mandatory 
component connected to an OR variation point. Furthermore, 
our approach can still be useful for small- and some medium- 
scale SPL architectures involving a short number of variants 
per Alternative variation points. In addition, the validity of 
the verification rules is shown with the theorems presented 
in Sect. 4. Such theorems validate the verification rules car- 
ried out by our framework for Mandatory, OR, Alternative, 
and Optional connections as well as for requires/excludes 
relationships. 

Finally, it should be noted that simply using proposi- 
tional formulas is not enough to validate the component 
interconnections of a product architecture are consistent 
with the component interconnections of an SPL archi- 
tecture. Component interconnections involve relationships 
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among component interfaces. However, our ontology-based 
approach has more expressive power to define and validate 
such relationships. 


6.5 Limitations and threats to validity 


Our work has a number of limitations. The process of 
manually partition both an SPL architecture and a product 
architecture can be a complex and error-prone task. Automat- 
ing such a process is not addressed by this paper and is 
regarded as future work. Our proposal is able to handle a 
large number of variants for OR variation points; however, 
this is not the case of Alternative variation points that can 
only support a short number of variants. Future work will 
look into improving the performance of OntoPAV for Alter- 
native variation points involving a larger number of variants. 
Although the Debugger is able to find the origin of errors, 
it does not allow us to make fixes on the fly. Further work 
is required to have a fully automated debugging process. As 
we have only addressed the verification of component inter- 
connections, further work is required to include support for 
quality of service (QoS) aspects. Although our framework 
supports requires and excludes constraints, it does not support 
complex constraints [47] involving arbitrary propositional 
formulas. Nevertheless, it is feasible to extend our frame- 
work to support complex constraints given that arbitrary 
propositional formulas can be easily handled by descrip- 
tion logic (DL) [31], which is employed by our framework. 
This is because description logic is more expressive than 
propositional logic. Such an extension to our framework con- 
cerns future work. In addition, our proposal does not support 
constraints with numerical features that require arithmetical 
operations. Incorporating these kinds of constraints is not 
straightforward and, thus, requires further work. 

Next, we discuss the validity threats that could affect the 
evaluation results. In the case of threats to internal validity, 
the results of the accuracy evaluation could be affected by a 
bias in the selection of the test cases assessed. We reduced this 
threat by using mutation testing, which allowed us to guide 
the generation of test cases to cover a wide range of points 
of failure. Regarding the results of the scalability evaluation, 
there are various factors that can have an impact on the exe- 
cution time of our system prototype such as the load imposed 
by operating system processes and other processes running 
on the machine. We repeated 30 times each experiment in 
order to reduce this threat. !7 

Threats to external validity are related to the fact that the 
artifacts evaluated may not reflect real world SPL architec- 
tures. In the case of the results of the accuracy evaluation, 
we improved external validity by using a test case, i.e., an 


'7 The repetitions were reduced to five times only when the Debugger 
degraded rapidly. 
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SPL architecture, that includes all the types of connections 
checked by our verifier, namely Mandatory, OR, Alterna- 
tive, and Optional connections as well as requires/excludes 
relationships. This threat was also diminished by testing all 
the mutation operators we defined. For this purpose, it was 
necessary to extend the SPL architecture in order to test the 
mutation operators 24 and 28 that were not covered by the 
original SPL architecture (see description of operators 24 and 
28 in Sect 6.2). Regarding the results of the scalability evalu- 
ation, both the SPL and product architectures were randomly 
generated in our experiments. This fact is a threat to exter- 
nal validity given that these architectures may not reflect real 
world SPL and product architectures. The factors that impact 
scalability are the number of requires/excludes relationships, 
Mandatory elements (i.e., components and connectors) as 
well as the number of variation points (i.e., optional compo- 
nents and optional connectors) and variants that are included 
in the SPL architecture. Hence, we reduced this threat by 
considering a varying number of requires/excludes relation- 
ships, Mandatory elements, variation points, and variants 
ranging from 9 to 44, 45 to 220, 162 to 792, and 810 to 3960, 
respectively (see Table 4). The total number of architecture 
elements that we considered in our experiments were 1026, 
1995, 3021, 4047, and 5016 (see Table 4). We also diminished 
this threat by considering worst case scenarios. For example, 
both scenario 1 and scenario 2 consider SPL architectures, in 
which Alternative variation points were connected consecu- 
tively in line (one to another). This can cause that terms of 
Rule 3 grow exponentially, as previously discussed. Lastly, 
scenario 3 tested the scalability problems faced by Alterna- 
tive variation points when associated with a larger number of 
variants. 


7 Related work 


Several works have been carried to address SPL reverse engi- 
neering, some of them focus on feature models such as [48]. 
However, the majority of them have focused at the source 
code level [49]. A more generic approach [12] presents a 
unified framework to help the adoption of SPLs from legacy 
systems. The authors’ approach enables feature identifica- 
tion along with their constraints and the generation of new 
products. The proposed framework enables the possibility of 
adapting it to different artifact types and algorithms. Rubin 
et al. [13] propose a development framework that guides the 
process of reengineering an SPL architecture in terms of com- 
ponents and their connections. However, the SPL architecture 
extraction process has to be carried out manually. A few 
efforts have been done to reverse engineer UML models of 
software architecture product lines. For example, Assungao 
et al. [11] propose an approach to automatically extract 
UML class diagrams with features annotations. UML class 


diagrams are indeed useful for the design of SPL architec- 
tures. Nevertheless, ADL-based designs complement class 
diagrams with a higher level of abstraction making it eas- 
ier to reason about the architecture and communicating the 
design choices among stakeholders. However, to the best of 
our knowledge, currently there are not approaches able to 
automatically extract ADL-based SPL architectures, hence 
making it necessary to do it manually. 

A related problem to SPL extraction from legacy products 
involves the coevolution between SPLs and products where 
the SPL needs to be extracted or merged to make it consis- 
tent with changes introduced separately in the products. For 
instance, Schulze et al. [50, 51] propose a method to ensure 
consistency among artifacts whereby the initial version of a 
product, the modified version of a product within the SPL, 
and the modified version within the application domain are 
merged to a consistent version. This process can be useful 
for future efforts targeting software architecture artifacts. 

Several efforts have been carried out to achieve ver- 
ification of feature models [4, 14-18, 32, 52-60];these 
approaches commonly identify whether a feature model rep- 
resents at least one product and whether the model contains 
any errors such as contradictory feature relationships. A num- 
ber of works have used an ontology-based approach to verify 
feature models [14, 32, 53, 55]. In [14] a predicate-based 
ontology language is used for modeling and formalizing fea- 
ture models, and consistency is checked with Prolog. Guo et 
al. [53] propose an ontology-based formalization of feature 
models. In this work, changes in the model can be verified 
for consistency by employing a dependency matrix. Other 
ontology-based efforts have used description logic to ver- 
ify the consistency of feature models [32, 55]. In our work, 
we follow an ontology-based approach similar to Wang et 
al. [32]; however, we focus on verifying consistency of 
product architectures rather than feature models. Verifying 
consistency of SPL architectures has received some atten- 
tion [19, 24]. Czarnecki et al. define an approach to verify 
the well-formedness of an SPL architecture (i.e., an annotated 
model) by using OCL constraints. The approach ensures that 
a well-formed SPL architecture will derive in well-formed 
product architectures (i.e., template instances) for any valid 
feature configuration. For this purpose, the user has to man- 
ually define OCL constraints. In this approach, the authors 
assume the product architecture can be easily derived by a 
Template Preprocessor by simply removing those elements 
whose presence conditions, in the SPL architecture, evaluate 
to false. Also, the solution of our previous work allows for 
automating the generation of the product architectures [6] by 
defining the SPL in and ADL and providing the feature tree 
selections. However, none of these works helps to ensure that 
a reverse engineered SPL architecture is consistent with a set 
of legacy product architectures. One challenging problem is 
verifying a product architecture maintains consistency when 
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itis derived from dependent product lines (aka. multi-product 
lines [61]); this given that an evolution of an artifact in one 
product line can introduce inconsistencies in the product 
architectures of the other product lines. In [24], multi-view 
models with variability are checked for consistency. Other 
work considers verifying the consistency of SPL implemen- 
tations [20]. More specifically, this approach verifies the 
programs are type safe, i.e., absence of references to unde- 
fined elements such as classes and variables. Thiim et al. 
[62] and Kastner et al. [63] provide an approach to verify at 
the implementation level the composition of multiple prod- 
uct lines. Other efforts check the consistency between the 
SPL architecture and the feature model [21]. Janota et al. 
[64] and van der Storm et al. [65] present an approach that 
verifies the mappings between feature and component archi- 
tecture models. Asadi et al. [16] propose an approach to detect 
inconsistencies between goal models and feature models. 
However, only a few approaches have addressed the issue 
of checking consistency between the product architecture 
and the SPL architecture, but mainly focusing on behavioral 
aspects [22, 23]. Therefore, approaches are still missing for 
verifying structural aspects of the commonality and variabil- 
ity of a manually extracted SPL architecture with respect to 
the product architectures from which it is derived. 


8 Conclusion 


In this paper, we have presented the OntoPAV framework, 
an approach to verify consistency of component intercon- 
nection aspects between a product architecture of a legacy 
system and a reverse engineered SPL architecture. We rely on 
the use of ontologies to perform such a verification. Impor- 
tantly, the user does not need to have skills on ontologies 
to use our approach, rather, we make use of model-driven 
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techniques to automate the populated ontology generation 
process. We illustrated our approach via a motivating scholar 
system example that shows the validity of our solution. Our 
evaluation results show that our verifier has a high accuracy 
in detecting consistency errors between product architec- 
tures and an SPL architecture. Lastly, we have shown that 
by employing a partitioning strategy our framework is able 
to achieve a linear performance scalability for small- and, in 
some cases, medium-scale approaches. 

Future work regards automating the process of partition- 
ing SPL and product architectures. We will also look into the 
issue of improving the performance of our approach when 
handling Alternative variation points. Future improvements 
of our work also include extending OntoPAV to support not 
only the verification of consistency at the architecture level, 
but also at finer granularity levels (e.g., a code block [66]). 
We believe the preliminary results of our accuracy evalu- 
ation are encouraging and it is regarded as a future work 
using a synthetic data set to strengthen the accuracy evalu- 
ation. Finally, we will explore using our approach to verify 
changes introduced at runtime to SPLs. 
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Table 7 Full list of mutation operators 


Operator 


Example 


1. Remove a Mandatory component 

2. Change a Mandatory component 
to an Optional component 

3. Change the name of a 
Mandatory component 

4. Add an Optional component to 
an OR group 

5. Add an Optional component to 
an Alternative group 

6. Remove an Optional component 
from an OR group 

7. Remove an Optional component 
from an Alternative group 

8. Remove an Optional component 
and its associated variants 

9. Change the name of a variant 

0. Change an Optional component 

to a Mandatory component 
1. Change the name of an 
Optional component 
2. Change the instantiation 


relationship of an Optional comp 


3. Remove an OR group 


relationship 


4. Change OR group relationship 
to Alternative group relationship 

5. Remove an Alternative group 
relationship 

6. Change Alternative group 
relationship to OR group relation 

7. Add a connection between two 


Mandatory components 


8. Add a connection between a 
Mandatory component and an 


Optional component 


9. Add a connection between an 
Optional component and a 
Mandatory component 

20. Add a connection between two 

Optional components 

21. Remove a connection between 

two Mandatory components 


22. Remove a connection between 


Remove the Mandatory component c_student from the SPL architecture 

Convert the Mandatory component c_student to an Optional 
component in the SPL architecture 

The name of the Mandatory component 
c_student is changed to c_studentV2 

The Optional component vp_area is disconnected from c_ui and connected 
to the receptacle IdegreeOption of the component c_degree as another degree option 

The Optional component vp_area is removed and its associated variant c_area is 
added as an alternative of the Optional component vp_term 

Remove both the Optional component vp_thesis and its associated variant 
c_thesis from the OR group in the SPL architecture 

The variant c_single_term is removed from the Alternative group in the SPL 
architecture 

Remove both the variation point vp_area and its associated variant c_area 
from the SPL architecture 

Change the name of the variant c_thesis by c_thesisvV2 

The Optional component vp_area and its associated variant c_area are transformed to 
a Mandatory component and the requires/excludes relationships are eliminated 

The name of the variant c_area is changed to c_areaV2. Another 
example is changing the variant c_thesis to c_thesisV2 

The variants c_thesis and c_article are interchanged so that now they instantiate 
the Optional components vp_article and vp_thesis, respectively 

The Optional components vp_professionalPractice, vp_article, 
and vp_thesis are removed along with their associated variants c_professionalPractice, 

c_article, and c_thesis 

The OR group relationship of our running example is transformed to an Alternative 
group relationship 

The Optional component vp_term is removed along with its variants 
c_single_termand c_multiple_term 

The Alternative group relationship of our running example is transformed to an OR 
group relationship 

A receptacle of the component c_student is connected to a facet of the component 
c_degree 

A receptacle of the component c_course is connected to a facet of the Optional 


component vp_area 


Connect a receptacle of the Optional component 
vp_area to the facet of the component 
c_course 

Connect a receptacle of the Optional component vp_area to a facet of the Optional 
component vp_term 


The connection between the components c_ui and c_course is removed 


The connection between the component c_ui and the Optional component vp_area 
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Table 7 continued 


Operator 


a Mandatory comp and Op comp 
23. Remove connection between Op 
comp and a Mandatory comp 
24. Remove a connection between 


two Optional components 
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Example 


is removed 

The Optional connection between the Optional component vp_area and the 
component c_course is removed 

Given that our running example does not count with such a type of connection, 


we added an Optional connection between the Optional component vp_term 


and the Optional component vp_area, and we considered this as the original SPL 


architecture for testing the operator. Then, after applying this operator, this 


connection is removed to obtain the mutant. 


25. Change the connection of a 
Mandatory receptacle with another 
Mandatory receptacle 

26. Change the connection direction 
between two Mandatory comp 

27. Change the connection direction 
between a Mandatory component 
and an Optional component 

28. Change the connection direction 


between two Optional components 


The receptacles Isubj ect and Icourse of the component c_ui 


are connected to the facets Icourse and Isubject, respectively 


The connection between the components c_ui and c_course is changed so that a 
receptacle of the component c_course is now connected to a facet of the component c_ui 
Change the connection between the component c_ui and the Optional component 


vp_area so that a receptacle of vp_area is connected to a facet of c_ui 


Given that our running example does not count with such a type of connection, we 


added an Optional connection between the Optional component vp_term 


and the Optional component vp_area (see operator 24). 


29. Change the name of a 
receptacle interface 

30. Change the name of a facet 
interface 

31. Add a requires relationship 

32. Remove a requires relationship 

33. Add an excludes relationship 


34. Remove an excludes relationship 


The name of the receptacle Icourse is changed to IcourseV2 


The name of the facet Isubj ect is changed to IsubjectV2 


We add a requires relationship from variant c_thesis to variant c_single_term 
The requires relationship from variant c_single_term to the variant c_area is removed 
We add an excludes relationship from variant c_thesis to the variant c_single_term 


The excludes relationship from variant c_multiple_term to the variant c_area is removed 
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